\hypertarget{index_sec_intro}{}\section{Introduction}\label{index_sec_intro}
Iris is a cycle level simulator technology developed for multi-\/core and multi-\/node networks. The simulator is based on the Manifold simulator framework. This document provides details of the simulator architecture and breakdown of components that include routers, interfaces, packet generators, network links and memory controllers. Details of the router micro-\/architecture, network topologies and memory controller design policies etc. are given in subsequent sections. Iris is released under a xxx open source license.\hypertarget{index_sec_directory}{}\section{Directory Structure}\label{index_sec_directory}

\begin{DoxyItemize}
\item The Iris source code is organized into the following main sub-\/directories: \par
 \par
  \par

\end{DoxyItemize}


\begin{DoxyItemize}
\item Topology: This directory contains the implementation of the supported topologies in Iris. Based on the inputs in the config file, the type of topology, number of nodes, number of MCs and other parameters are identified. Objects of each type are instantiated and its connections are generated. Connections include those between the interface and terminal node, interfaces and routers and between routers.
\end{DoxyItemize}


\begin{DoxyItemize}
\item Components: This directory contains implementations of all the components and their functionalities in an Iris network. Components include routers, interface, packet generators, memory controllers etc. A detailed explanation of each of these components is given below.
\end{DoxyItemize}


\begin{DoxyItemize}
\item Data\_\-types: The classes in this directory define the data types for modeling the units of transmission in an Iris network. The 3 main components here are: network packet, network flit and link data.
\end{DoxyItemize}


\begin{DoxyItemize}
\item Interfaces: This directory defines the interfaces for the basic components of Iris such as the Interface, Router and Terminal. It also includes the genericHeader.h file that contains some enum definitions for fields such as routing, arbitration and destination distribution schemes.
\end{DoxyItemize}


\begin{DoxyItemize}
\item Examples: The files in this directory are examples of some topologies that are currently supported in Iris. The entry point of execution is in the main() function of these files. Functions to parse the config file and dump out the output of the execution are included here. The examples also illustrate how to instantiate the Manifold kernel, the topology and the components. It also contains the ref\_\-out directory which contains the reference outputs from runs on certain configurations.
\end{DoxyItemize}


\begin{DoxyItemize}
\item Test: This directory contains the unit test cases for all the components of Iris. When a new component is added to Iris, its corresponding unit test case must be added here.
\end{DoxyItemize}\hypertarget{index_sec_overview}{}\section{Overview}\label{index_sec_overview}
The Iris simulator takes input in the form of a config file present in the /config directory, identifying the type of topology and number of nodes.

Features of the Iris simulator:
\begin{DoxyEnumerate}
\item Supports sequential and parallel execution modes.
\item Support for k-\/ary 2D torus, two-\/node and ring topologies -\/ Support for other topologies like mesh, fat tree and flattened butterfly are planned for future work.
\item Supports multiple operating systems -\/ Tested on Red Hat Enterprise Linux 5.5 and Ubuntu 10.10. Plan to support more OS in the future.
\item The network is modeled as a request-\/reply network.
\end{DoxyEnumerate}

Below is a brief description of the components of the Iris simulator. All of these components are defined in the /trunk/code/models/iris/components directory.

1. Router: This component is defined in the simpleRouter.h class. This version of Iris has a simple 5 stage pipeline router. The 5 stages are: Buffer Write (BW), Route Computation (RC), Virtual Channel Allocation (VCA), Switch Allocation (SA) and Switch Traversal (ST).
\begin{DoxyItemize}
\item A detailed diagram of the router architecture and associated data structures is given below: \par
  \par

\end{DoxyItemize}

The APIs that deal with each stage of the pipeline are as indicated below in the code snippets:


\begin{DoxyCode}
 void SimpleRouter::do_input_buffering(HeadFlit* hf, uint inport, uint invc) 
\end{DoxyCode}

\begin{DoxyItemize}
\item {\itshape hf\/} is the fead flit of the incoming packet
\item {\itshape inport\/} is the input port on which the flit arrives
\item {\itshape invc\/} is the input virtual channel on which the flit arrives
\end{DoxyItemize}


\begin{DoxyCode}
 void GenericRC::push (Flit* f, uint ch ) 
\end{DoxyCode}

\begin{DoxyItemize}
\item {\itshape f\/} is the flit pointer of the flit that is pushed on the buffer
\item {\itshape ch\/} is the channel on which it comes in \par
 This is the stage where route computation is performed. The functions for this are present in genericRC.h. The push function is invoked during the input buffering stage on a flit and when it gets buffered.
\end{DoxyItemize}


\begin{DoxyCode}
 void SimpleRouter::do_vc_allocation() 
\end{DoxyCode}

\begin{DoxyItemize}
\item This is the next stage in the pipeline. Output virtual channel allocation happens at this stage. Only head flits need to go through this stage.
\end{DoxyItemize}


\begin{DoxyCode}
 void SimpleRouter::do_switch_allocation()
\end{DoxyCode}

\begin{DoxyItemize}
\item This is the API for stage number 4 in the pipeline. Switch allocation happens at this stage. All the flits of a packet need to go through this phase.
\end{DoxyItemize}


\begin{DoxyCode}
 void SimpleRouter::do_switch_traversal() 
\end{DoxyCode}

\begin{DoxyItemize}
\item This is the final stage of the pipeline. The crossbar is configured for corresponding pairs of input and output ports.
\end{DoxyItemize}

2. Packet Generator: The packet generator is a simple stochastic packet generator defined in the pktgen.h class that acts as an end terminal to the Iris network. It can generate packets as well as receive packets from other processors. An end terminal in Iris uses a packet level credit based approach for flow control. Some of the tunable parameters in the packet generator class are packet injection time and destination ID. The packet generator class uses the GNU Scientific Library (GSL) for generating these values. \par
 The packet generator picks the destination based on some condition given by the user. The destination node ID is determined by either manipulating the bits or sending a request to the memory controller depending on user input.

\par
 The bit manipulation currently supports 2 types. The first one is where the destination ID is the bit reversal of the source ID. The API for that is as shown below:


\begin{DoxyCode}
 uint PktGen::get_bit_rev_dest(uint b) 
\end{DoxyCode}

\begin{DoxyItemize}
\item {\itshape b\/} is the source node ID.
\end{DoxyItemize}

\par
 The second type of destination is obtained by using the formula: 
\begin{DoxyCode}
 tmp = (no_nodes-2+node_id)%no_nodes; 
\end{DoxyCode}

\begin{DoxyItemize}
\item {\itshape node\_\-id\/} is the source node ID.
\end{DoxyItemize}

\par
 In the case of sending a request to an MC, the following code is invoked: 
\begin{DoxyCode}
 tmp = mc_positions.at(gsl_rng_get(dst_gen)%no_mcs); 
\end{DoxyCode}

\begin{DoxyItemize}
\item {\itshape mc\_\-positions\/} is a vector containing the position of all MCs in the grid.
\end{DoxyItemize}

3. Interface: This component models the network interface between the processor and the router within the Iris network. This component converts the packet to a format that enables it to be transferred on the network between components. In other words, there is conversion from flits to phits and vice versa at this stage. It is defined in the ninterface.h class.

4. Memory Controller: This component models the memory controller that accepts a request from a processor (request for a cache line, for example) and returns the reply after some fixed amount of DRAM latency. It is defined in the simpleMC.h class.

5. Route Computation Unit: This unit is a sub-\/component of the router architecture. It is defined as a separate class in genericRC.h to decouple the various routing algorithms from the router class. The class provides routing functions for various topologies supported in Iris.

6. Virtual Channel Allocation Unit: This sub-\/component is defined in genericVcAllocator.h and implements the class for virtual channel arbitration. Each packet's head flit has to be assigned an output virtual channel after it is buffered in a router. Only the head flits go through this phase.

7. Switch Arbitration Unit: Another sub-\/component of the router, this class is defined in the genericSwitchArbiter.h file. This class deals with the switch arbitration for each output port i.e, from all the contending packets, a winner is selected for each output port to send the flits out. Once the winner is picked, the crossbar is configured for the corresponding pairs of input and output ports. Various schemes have been implemented for arbitration including, round robin, round robin with priority and FCFS. All the flits of a packet go through this phase.

8. Buffer: This is the basic buffering unit of the router modeled in the genericBuffer.h file. It represents the physical buffers that are present on a router and can be divided into multiple virtual channels. A credit based switching scheme is implemented to ensure that adequate buffer space is present in any component before packets are transmitted.\hypertarget{index_sec_using}{}\section{Using Iris}\label{index_sec_using}
\hypertarget{index_download}{}\subsection{Download}\label{index_download}
To inspect the Iris source code, visit the following link: \par
 
\begin{DoxyItemize}
\item \href{https://svn.ece.gatech.edu/repos/Manifold/trunk/code/models/network/iris/}{\tt https://svn.ece.gatech.edu/repos/Manifold/trunk/code/models/network/iris/}
\end{DoxyItemize}\hypertarget{index_installation}{}\subsection{Installation}\label{index_installation}
Before installing Iris, the pre-\/requisite is to install the Manifold kernel. Please refer to the Manifold spec for steps on how to proceed with this step.\hypertarget{index_externallibraries}{}\subsection{External Library Dependencies}\label{index_externallibraries}
Iris depends on the following external libraries. Ensure that these libraries are installed before compiling Iris.
\begin{DoxyEnumerate}
\item GSL (GNU Scientific Library) version 1.14 used by the packet generators: This library can be downloaded from the link 
\begin{DoxyItemize}
\item \href{http://www.gnu.org/software/gsl/}{\tt http://www.gnu.org/software/gsl/} 
\end{DoxyItemize}
\item open MPI version 1.4.3 for running the parallel version of Iris: This can be downloaded from 
\begin{DoxyItemize}
\item \href{http://www.open-mpi.org/software/ompi/v1.4/}{\tt http://www.open-\/mpi.org/software/ompi/v1.4/} 
\end{DoxyItemize}
\item CPPUNIT testing framework version 1.10.2: This is used to write the unit test cases for the components of Iriss and can be found at 
\begin{DoxyItemize}
\item \href{http://sourceforge.net/projects/cppunit/}{\tt http://sourceforge.net/projects/cppunit/} 
\end{DoxyItemize}
\end{DoxyEnumerate}\hypertarget{index_tocompile}{}\subsection{To compile:}\label{index_tocompile}
\par
 To compile the Iris source code and generate the Iris library comprising all the components and topologies, follow the steps given below: \par

\begin{DoxyItemize}
\item Make sure you have the gnu cpp compiler (version 4.1.2, atleast) installed. The following steps will compile the simulator.
\item Untar the source
\item tar -\/zvxf $<$tar\_\-name$>$
\item cd /trunk/code/models/iris/
\item ./configure
\item make
\end{DoxyItemize}

The makefiles are generated using gnu automake tools (version 1.9.6)  \href{http://www.gnu.org/software/automake/}{\tt http://www.gnu.org/software/automake/} .The simulator has been tested using gcc version 4.1.2 on the Ubuntu and Red Hat distributions of Linux. It can be compiled as a 64 bit binary. \par


configure.ac and Makefile.am are the files to change if you need to modify the compiler flags. You may also need to rerun aclocal, autoconf and automake if the changes are not reflected in the make. \par


For example: If you do not wish to make use of the GSL library for use in the packet generator, you can remove the -\/lgsl flag from the Makefile in the examples directory in this line


\begin{DoxyCode}
 LDFLAGS += -lgsl -lgslcblas -L$(IRISLIB) -lmanifold ../src/iris/libiris.a 
\end{DoxyCode}


The next few steps will tell you how to get started with an example.\hypertarget{index_howto}{}\subsection{To Execute}\label{index_howto}
To use Iris to run a simulation on a particular topology, 2 main steps are involved.
\begin{DoxyEnumerate}
\item Writing a config file for that particular topology and adding it to the config directory.
\item Writing a program to parse the config file, generate the topology and run the simulation. This file is added to the examples directory. A corresponding entry should also be made to the makefile in the examples directory to generate an executable.
\end{DoxyEnumerate}

Examples of how to do the above are given in the next section. Compile the examples directory to include the new files.

To run Iris in sequential mode (single threaded), the following command is executed on the command line. manifold\_\-node\_\-mpi is a sample executable that is present in the examples directory. \par

\begin{DoxyItemize}
\item ./manifold\_\-node\_\-mpi config/manifold.cfg \par
 To run Iris in parallel (multi threaded), run the following command on the command line.
\item mpirun -\/np 2 ./manifold\_\-node\_\-mpi config/manifold.cfg \par
 Where manifold.cfg is the config file which acts as input to the simulator. The parameters that are given in the input config file are:
\item Type of topology
\item Number of nodes
\item Grid size
\item Number of memory controllers
\item Number of ports on each router
\item Number of virtual channels
\item Buffer width
\item Number of credits
\item Route computation method
\item Response payload length
\item Memory latency
\item Mean IRT
\item Link width
\item Maximum simulation time
\end{DoxyItemize}

Values are given for each of these parameters and are taken as inputs for the simulation.

\par
 The code that parses the config file is present in the following function of the topology.cc file in the topology folder. This function has to be modified to add any new parameters that may be added to the config file: 
\begin{DoxyCode}
 void Topology::init(std::string conf_file)
\end{DoxyCode}

\begin{DoxyItemize}
\item {\itshape conf\_\-file\/} is the input configuration file
\end{DoxyItemize}\hypertarget{index_example}{}\subsection{Example}\label{index_example}
For example, if the user wants to test a 16 node, 2d torus with 4 virtual channels and standard 5 port routers, the config files looks something like this (This config file is present in the examples/config directory as torus.cfg):


\begin{DoxyItemize}
\item iris:noc\_\-topology TORUS
\item iris:no\_\-nodes 16
\item iris:grid\_\-size 4
\item iris:no\_\-mcs 0
\item iris:no\_\-ports 5
\item iris:no\_\-vcs 4
\item iris:int\_\-buff\_\-width 4
\item iris:credits 3gcc version 4.1.2
\item iris:rc\_\-method TORUS\_\-ROUTING
\item mc:resp\_\-payload\_\-len 256
\item mc:memory\_\-latency 100
\item iris:mean\_\-irt 50
\item iris:link\_\-width 128
\item iris:max\_\-simtime 1000000
\end{DoxyItemize}

Then the simulator is invoked using the command given in section To Execute. This has example has been tested on gcc version 4.1.2.

An example implementation of the program to create and run the simulation on a topology is as given below. This program is the manifold\_\-node\_\-mpi.cc file in the examples directory.


\begin{DoxyCode}
 /*
 ================================================================================
      =====

       Filename:  manifold_node.cc

    Description:  This is a simple frontenf for using IRIS with pktgen and
    routers for diff topologies
    Can run either single threaded or on two logical processes using MPI.

        Version:  1.0
        Created:  04/20/2010 02:59:08 AM
       Revision:  none
       Compiler:  mpicxx

         Author:  Mitchelle Rasquinha (), mitchelle.rasquinha@gatech.edu
        Company:  Georgia Institute of Technology

 ================================================================================
      =====
 */

#include        "iris/irisTerminal.h"
#include        "iris/irisInterface.h"
#include        "iris/irisRouter.h"
#include        "iris/topology/topology.h"

#include        "kernel/clock.h"
#include        "kernel/manifold.h"
#include        "kernel/component.h"

#include        "iris/components/pktgen.h"

#include        "iris/topology/topo_creator.h"

#include        "mpi.h"
#define         TOT_PROCS 2

using namespace manifold::kernel;

int Mytid; //task id
FILE* log_file;

Clock* master_clock = new manifold::kernel::Clock(1);

int
main ( int argc, char *argv[] )
{
    Manifold::Init(argc,argv);

    //make sure only 2 tasks are started.
    if( TOT_PROCS != manifold::kernel::TheMessenger.get_node_size()) {
        cerr << "Please specify \"-np 2\" for mpirun!" << endl;
        return 1;
    }

    MPI_Comm_rank(MPI_COMM_WORLD, &Mytid);

    ullint sim_start_time = time(NULL);
    if(argc<2)
    {
        cout << "Error: Requires config file for input parameters\n";
        return 1;
    }

    //topology creator creates network interfaces and routers and connect them.
    Topology* tp;
    try {
        tp = TopoCreator :: create_topology(argv[1]);
    }
    catch (CreateTopologyException e) {
        cout << e.what() << endl;
        exit(1);
    }

    //create PktGens and connect them with network interfaces.
    for(int i=0; i<tp->no_nodes; i++) {
        CompId_t id = Component::Create<PktGen>(0, i, 4, tp->no_nodes, 50);
        IrisTerminal* term = dynamic_cast<PktGen*>(Component::GetComponent(id));
        if(term)
            tp->connect_terminal(term);
    }

    //MPI_Barrier(MPI_COMM_WORLD);   // All components have finished Init
    std::cerr << "\n############# Start Timing Simulation for " << tp->max_simtim
      e << " cycles Proc:"<< Mytid;

    Manifold::StopAt(tp->max_simtime);
    Manifold::Run();

    tp->print_noc_stats();
    //MPI_Barrier(MPI_COMM_WORLD);
    if ( Mytid == TOT_PROCS-1 )
    {
        ullint sim_time_ms = (time(NULL) - sim_start_time);
        cerr << "\n\n************** Simulation Stats ***************\n";
        cerr << " Simulation Time: " << sim_time_ms << endl;
    }

    MPI_Finalize();
}
\end{DoxyCode}
\hypertarget{index_output}{}\subsection{Output}\label{index_output}
The output that appears on the console after Iris completes execution consists of the following:
\begin{DoxyItemize}
\item Dump of the configuration 
\begin{DoxyCode}
 ************* IRIS CONFIG KNOB **************
credits 3
grid_size       4
int_buff_width  4
link_width      128
mean_irt        50
memory_latency  100
no_mcs  0
no_nodes        16
no_ports        5
no_vcs  4
noc_topology    TORUSgcc version 4.1.2 
rc_method       TORUS_ROUTING
resp_payload_len        256
\end{DoxyCode}

\end{DoxyItemize}


\begin{DoxyItemize}
\item Per packet generator output stats 
\begin{DoxyCode}
 PktGen[2] packets_out: 2450
 PktGen[2] packets_in: 2449
 PktGen[2] avg_latency: 21.1666
 PktGen[2] stat_last_packet_out_cycle: 99996
 PktGen[2] stat_last_packet_in_cycle: 99969
\end{DoxyCode}

\end{DoxyItemize}


\begin{DoxyItemize}
\item {\itshape packets\_\-out\/} is the total number of packets sent by this generator out on the network
\item {\itshape packets\_\-in\/} is the total number of packets received by this terminal from other sourcs
\item {\itshape avg\_\-latency\/} is the average number of cycles taken by packets to reach this node
\item {\itshape stat\_\-last\_\-packet\_\-out\_\-cycle\/} is the last cycle of the simulation in which a packet was sent out on the network
\item {\itshape stat\_\-last\_\-packet\_\-in\_\-cycle\/} is the last cycle of the simulation in which a packet was received on this terminal
\end{DoxyItemize}


\begin{DoxyItemize}
\item Per router output stats 
\begin{DoxyCode}
 SimpleRouter[2] packets_in: 9797
 SimpleRouter[2] packets_out: 9796
 SimpleRouter[2] flits_in: 19594
 SimpleRouter[2] flits_out: 19592
 SimpleRouter[2] avg_router_latency: 4.08085
 SimpleRouter[2] last_pkt_out_cy: 99966
 SimpleRouter[2] per_port_pkts_out: 2449 0 0 0 2449 0 2449 0 0 0 2449 0 0 0 0 0 0
       0 0 0 
 SimpleRouter[2] per_port_avg_lat: 4 0 0 0 4.3234 0 4 0 0 0 4 0 0 0 0 0 0 0 0 0 
 SimpleRouter[2] per_port_last_pkt_out_cy: 99966 0 0 0 99956 0 99961 0 0 0 99959 
      0 0 0 0 0 0 0 0 0 
\end{DoxyCode}

\end{DoxyItemize}


\begin{DoxyItemize}
\item {\itshape packets\_\-in\/} is the total number of packets received on the router
\item {\itshape packets\_\-out\/} is the total number of packets sent out on this router
\item {\itshape flits\_\-in\/} is the total number of flits received on the router
\item {\itshape flits\_\-out\/} is the total number of flits sent out on this router
\item {\itshape avg\_\-router\_\-latency\/} is the average number of cycles a packet has to stay in this router before being sent out completely
\item {\itshape last\_\-pkt\_\-out\_\-cy\/} is the last cycle of the simulation in which a packet was sent out on this router
\item {\itshape per\_\-port\_\-pkts\_\-out\/} is the number of packets out per port
\item {\itshape per\_\-port\_\-avg\_\-lat\/} is the average latency on each port of the router in cycles
\item {\itshape per\_\-port\_\-last\_\-pkt\_\-out\_\-cy\/} is the last cycle of the simulation in which a packet was sent out on this port
\end{DoxyItemize}\hypertarget{index_faq}{}\section{Frequently Asked Questions}\label{index_faq}
1. How is a new topology created?
\begin{DoxyItemize}
\item To add a new topology, create the header and source files for the new topology in the /iris/topology directory. Ensure that the new topology class derives from Topology class defined in /iris/topology/topology.h and implements the pure virtual functions. Also remember to add an entry for the new topology in Makefile.am in iris directory so it gets compiled.
\end{DoxyItemize}

2. Adding a new routing function:
\begin{DoxyItemize}
\item To add a new routing function, add a new routing function to the genericRC.h file and its corresponding implementation in genericRC.h. Add an entry for the new routing function in the enum ROUTING\_\-SCHEME in the /iris/interfaces/genericHeader.h file. The new routing function can be invoked by changing the config file to use the new function against the rc\_\-method parameter.
\end{DoxyItemize}

3. Adding a new router:
\begin{DoxyItemize}
\item To add a new type of router to Iris, create the header and source files in /iris/components directory. The new router must derive from the IrisRouter class defined in iris/interfaces directory. A new enum maybe defined to add the new type of router in genericHeader.h. A corresponding field is to be added in the config file as well as some changes in the frontend file (examples directory) would also be required.
\end{DoxyItemize}

4. Adding new fields to the packet:
\begin{DoxyItemize}
\item Add the new field to the file /iris/data\_\-types/networkPacket.h Example: unsigned int transaction\_\-id;
\end{DoxyItemize}


\begin{DoxyItemize}
\item Add the new field to the corresponding part in the head/body/tail of the flit packet as defined in /iris/data\_\-types/flit.h Example: unsigned int transaction\_\-id; // added to class HeadFlit
\end{DoxyItemize}


\begin{DoxyItemize}
\item To use this field, include code as shown in sampleator has been tested using gcc version 4.1.2 on the Ubuntu and Red Hat distributions of Linux. It can be compiled as a 64 bit binary. Configure.acÂ and Makefile.am are the files to change if you need to modify the compiler flags. You may also need to rerun aclocal, autoconf and automake if the changes are not reflected in the make. For example: If you do not wish to make use of the GSL librar below from /iris/components/simpleRouter.cc file which uses hop\_\-count in the function SimpleRouter::do\_\-input\_\-buffering(HeadFlit$\ast$ hf, uint inport, uint invc) Example: hf-\/$>$transaction\_\-id = 1000;
\end{DoxyItemize}

5. How to add a new destination distribution type to the packet generator?
\begin{DoxyItemize}
\item To do so, add a new function to the pktgen.h class and its corresponding implementation in pktgen.cc. Add the condition to invoke the new type of distribution. Also, add the new type to the enum DEST\_\-DISTRIBUTION\_\-TYPE defined in genericHeader.h. 
\end{DoxyItemize}